Using Difficulty of Prediction to Decrease Computation: Fast Sort, Priority Queue and Convex Hull on Entropy Bounded Inputs

There is an upsurge in interest in the Markov model and also more general stationary ergodic stochastic distributions in theoretical computer science community recently (e.g. see [Vitter,KrishnanSl], [Karlin,Philips,Raghavan92], [Raghavan9 for use of Markov models for on-line algorithms, e.g., cashing and prefetching). Their results used the fact that compressible sources are predictable (and vise versa), and showed that on-line algorithms can improve their performance by prediction. Actual page access sequences are in fact somewhat compressible, so their predictive methods can be of benefit. This paper investigates the interesting idea of decreasing computation by using learning in the opposite way, namely to determine the difficulty of prediction. Tha t is, we will a p proximately learn the input distribution, and then improve the performance of the computation when the input i s not too predictable, rather than the reverse. To our knowledge, this is first case of a computational problem where we do not assume any particular fixed input distribution and yet computation is decreased when the input is less predictable, rather than the reverse. We concentrate our investigation on a basic computational problem: sorting and a basic data structure problem: maintaining a priority queue. We present the first known case of sorting and priority queue algorithms whose complexity depends on the binary entropy H 5 1 of input keys where assume that input keys are generated from an unknown but arbitrary stationary ergodic source. This is, we assume that each of the input keys can be each arbitrarily long, but have entropy H. Note that H can be estimated in practice since the compression ratio p using optimal Ziv-Lempel compression limits t o 1/H for large inputs. Although sets of keys found in practice can not be expected to satisfy any fixed particular distribution such as uniform distribution, there is a large well documented body of empirical evidence that shows this compression ratio p and thus 1 / H is a constant for realistic inputs encountered in practice [l, 311, say typicall around 3 to a t most 20. Our algorithm runs in O(nlog($)) sequential expected time to sort n keys in a unit cost sequential RAM machine. This is O(n log log n) with the very reasonable assumption that the compression *Email addresses: reifQcs.duke.edu and chenQcs.duke.edu. S u p ported by DARPA/ISTO Contracts N00014-8SK-0458, DARPA N00014-91-J-1985, N00014-91-C-0114, NASA subcontract 550-63 of prime contract NAS5-30428, US-Israel Binational NSF Grant 8800282/2, and NSF Grant NSF-IRI-91-00681. 0272-5428/93 $03.00